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Program Description^ 


Literacy Express is a preschool curriculum designed for three- 
to five-year-old children. It is structured around units on oral 
language, emergent literacy, basic math, science, general knowl- 
edge, and socioemotional development. It can be used in half- 
or full-day programs with typically developing children and 


children with special needs. It provides professional develop- 
ment opportunities for staff; teaching materials; suggested 
activities; and recommendations for room arrangement, daily 
schedules, and classroom management. 


Research^ 


Three studies of Literacy Express that fall within the scope of the 
Early Childhood Education review protocol meet What Works 
Clearinghouse (WWC) evidence standards. The three studies 
include 1,004 preschool children from three to five years of age 
from 70 preschools in Florida and California.'* 


medium to large for oral language, print knowledge, and phono- 
logical processing and small for cognition and math. No studies 
that meet WWC evidence standards with or without reservations 
examined the effectiveness of Literacy Express on preschool 
children in the early reading and writing domain. 



Based on these three studies, the WWC considers the extent 
of evidence for Literacy Express on preschool children to be 



1. This report has been updated to include a review of two studies that were released since 2007. These studies are within the scope of the protocol and 
meet evidence standards. The findings described in the previous Literacy Express intervention report were based, in part, on studies by Lonigan (2005, 
2006). A review of those studies for the present report revealed that they were based on a second cohort of children from a randomly assigned set of 
preschools. Since children entered the preschools after random assignment, the initial equivalence of the treatment and control groups must be estab- 
lished. Tests of the equivalence of the analysis samples conducted by Lonigan showed statistically significant differences between the Literacy Express 
group and the control group on 5 of 11 outcome measures. Hence, results from the Lonigan (2005, 2006) studies were not considered when preparing 
the present intervention report. A complete list and disposition of all studies reviewed are provided in the references. 

2. The descriptive information for this program was obtained from publicly available sources: the research literature (Lonigan, Farver, Clancy-Menchetti, & 
Phillips, 2005) and from the developer as part of the WWC’s standard developer contact process. The WWC requests developers to review the program 
description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program is beyond 
the scope of this review. The literature search reflects documents publicly available by May 2009. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Handbook, Chapter III) 
as described in protocol Version 2.0. 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 
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Eff6CtiV6n6SS Literacy Express was found to have positive effects on orai ianguage, print knowledge, and phonoiogicai processing and no discernibie 
effects on cognition and math for preschooi children. 





Oral 

language 


Print 

knowledge 


Phonological 

processing 


Early 

reading/ 

writing 


Cognition 


Math 


Rating of 
effectiveness 


Positive effects 


Positive effects 


Positive effects 


na 


No discernible 
effects 


No discernible 
effects 


Improvement 


Average: +^2 


Average: +15 


Average: +12 


na 


Average: +1 


Average: 0 


index® 


percentile points 


percentile points 


percentile points 




percentile point 


percentile points 




Range: -2 to 
+23 percentile 
points 


Range: +2 to 
+24 percentile 
points 


Range: +6 to 
+21 percentile 
points 


na 


Range: -5 to 
+5 percentile 
points 


Range: -1 to 
+2 percentile 
points 

na = not applicable 



Absence of conflict The PCER Consortium (2008) study summarized in this interven- 
Of interest fion report had numerous contributors, inciuding staff of Math- 
ematica Poiicy Research. Because the principai investigator for 
the WWC Eariy Childhood Education review is also a Mathematica 



staff member, the study was rated by Chesapeake Research 
Associates, which aiso prepared the intervention report. The 
report was then reviewed by the principal investigator, a WWC 
Quality Assurance reviewer, and an externai peer reviewer. 



Additionai program 
information 



Developer and contact 

Developed by Christopher J. Lonigan, Ph.D.; Jeanine Clancy- 
Menchetti, Ph.D.; Beth M. Phillips, Ph.D.; and colleagues, 
Literacy Express is currently available in limited commercial 
distribution through Literacy Express. Email: LiteracyExpress® 
hotmail.com. 

Scope of use 

Literacy Express has been implemented by preschool and child 
care programs in California, Florida, Massachusetts, New Mexico, 
and Texas. These programs have included typically developing 
children, children with special needs, and English language 
learners. However, information is not available on the number 
or demographics of children or centers using this program. 



Teaching 

Literacy Express is a preschool curriculum intended to improve 
children’s language development and early literacy achievement 
that can be implemented in various early childhood settings. It 
includes daily individual, small-group, and large-group activities 
and a balance of teacher-initiated and child-initiated activities. 
Literacy Express is structured around 10 thematic units, covered 
in three to four weeks each, that can be integrated into class- 
room activities.® The units are sequenced in order of increasing 
complexity, each building on the previous one. In each unit, 
teachers use three specific and brief small-group activities 
daily (dialogic reading, phonological awareness activities, and 
print knowledge activities) with groups of three to five children. 
Teachers also select from a number of suggested large-group 



5. These numbers show the average and range of student-level Improvement Indices for all findings across the studies. 

6. The research reviewed for this report is based on an earlier version of the curriculum with 11 thematic units. 
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Additional program activities for each unit that involve active child participation to 
information (continued) use skills learned in the small-group activities in new contexts. 

The curriculum includes both teacher- and child-directed math 
and science activities that follow a developmental scope and 
sequence. The curriculum also provides suggested activities— 
art, cooking, science, discovery, and gross and fine motor 
activities— for many independent play centers. The complete 
curriculum package includes a teacher’s manual; 10 thematic 
unit guides; a unit guide for augmentative or summer activities 
(“Off to Kindergarten”); and key curriculum materials such as 
shape materials, more than 80 thematically linked picture books 



and alphabet books, and phonological awareness activity picture 
cards, letters, and numbers. Teachers participate in a two-day 
professional development workshop at the beginning of the 
school year and in either two additional half-day workshops 
or one additional full-day workshop. 

Cost 

The current cost for a complete Literacy Express classroom 
package is $2,300. Professional development fees vary by 
the size of the group and the number of trainers. 



RSSSarch Four studies reviewed by the WWC investigated the effects of 
Literacy Express on preschool children. Three studies (Farver, 
Lonigan, & Eppe, 2009; Lonigan, Farver, Clancy-Menchetti, & 
Phillips, 2005; and PCER Consortium, 2008) are randomized 
controlled trials that meet WWC evidence standards. The 
remaining study does not meet WWC evidence standards 
or eligibility screens. 

Meets evidence standards 

Farver, Lonigan, and Eppe (2009) randomly assigned 96 
Spanish-speaking children in 10 classes in a Los Angeles Head 
Start center to one of three groups; (1) an English-only Literacy 
Express group, (2) a transitional Literacy Express group in which 
instruction began in Spanish and transitioned to English over 
the course of the intervention, and (3) a control group. This 
report focuses on the comparison of English language outcomes 
between the combined English-only and transitional groups and 
the no-treatment comparison group. For both Literacy Express 
groups, the intervention was implemented in small groups in a 
classroom adjacent to the children’s regular classroom. Pretest 
and posttest data were obtained for 94 children (31 English- 
only Literacy Express, 31 transitional Literacy Express, and 32 
control). On average, the participants were age 54.5 months (4.5 



years); 54% were boys. The study investigated oral language, 
phonological awareness, and print knowledge. The control 
condition was the High/Scope curriculum. 

Lonigan et al. (2005) randomly assigned 18 preschools (mostly 
Head Start centers) in Florida and 30 preschools in California to 
implement Literacy Express or to a control group. Pretest and 
posttest data (collected in the fall and spring of the preschool 
year) were obtained for 722 children (486 Literacy Express and 
236 control). Fifty-one percent of the children were male; 35% 
were Hispanic, 8% were Caucasian, and 56% were African 
American; and 52% of the children in the California preschools 
and 1% of the children in the Florida preschools were Spanish- 
speaking English language learners. All children were considered 
at risk for academic difficulties as determined by pretest scores 
on a measure of cognitive performance. The study investigated 
effects on oral language, print knowledge, phonological 
processing, and cognition. The control group implemented 
the preschool’s standard curriculum, which in most cases was 
High/Scope or Creative Curricuium®. 

PCER Consortium (2008) assessed the effectiveness of 
Literacy Express as part of the Preschool Curriculum Evalu- 
ation Research (PCER) effort.^ The PCER Consortium (2008) 
used a randomized controlled trial design in which 12 full-day 



7. The PCER Consortium (2008) evaluated a total of 14 preschool curricula, including Literacy Express, in comparison to respective controi conditions. 
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RGSCarch (continued) preschools in Florida were randomly assigned to implement 
Literacy Express or to a control group. Pretest and posttest 
data (collected In the fall and spring of the preschool year) were 
obtained for 188 children (93 Literacy Express and 95 control). 
Fifty-four percent of the children were male; 6% were Flispanic, 
30% were Caucasian, and 59% were African American; and 36% 
were reported to have a disability. The study investigated effects 
on oral language, print knowledge, phonological processing, 
and math. The control condition was the High/Scope curriculum. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 



Standards Flandbook, Appendix G). The extent of evidence 
takes into account the number of studies and the total sample 
size across the studies that meet WWC evidence standards 
with or without reservations.® 

The WWC considers the extent of evidence for Literacy 
Express to be medium to large for oral language, print knowledge, 
and phonological processing and small for cognition and math 
for preschool children. No studies that meet WWC evidence 
standards with or without reservations examined the effective- 
ness of Literacy Express in the early reading and writing domain 
for preschool children. 



Effectiveness Findings 

The WWC review of interventions for Literacy Express addresses 
student outcomes in six domains: oral language, print knowl- 
edge, phonological processing, early reading and writing, 
cognition, and math. The studies included in this report cover 
five domains: oral language, print knowledge, phonological 
processing, cognition, and math. The findings below present the 
authors’ estimates and WWC-calculated estimates of the size 
and the statistical significance of the effects of Literacy Express 
on preschool children.® 

Oral language. Farver, Lonigan, and Eppe (2009) analyzed 
the effectiveness of Literacy Express on oral language using 
the Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTOPPP) Receptive Vocabulary and Definitional 
Vocabulary subtests. The analyses showed, and the WWC 
confirmed, a statistically significant positive effect of 0.57 across 



the two measures (0.55 for receptive vocabulary and 0.60 for 
definitional vocabulary) when children in the Literacy Express 
group were compared to children in the control group. According 
to WWC criteria, this study shows a potentially positive effect 
on oral language. 

Lonigan et al. (2005) analyzed the effectiveness of Literacy 
Express on oral language using the Preschool Language Scales- 
IV (PLS-IV) Expressive Communication subtest. The authors 
found, and the WWC confirmed, a statistically significant and 
substantively important positive effect of 0.30 when children 
In the Literacy Express group were compared to children In the 
control group. According to WWC criteria, this study shows 
a potentially positive effect on oral language. 

The PCER Consortium (2008) analyzed the effectiveness 
of Literacy Express on oral language using the Peabody 
Picture Vocabulary Test III (PPVT-lll) and the Test of Language 



8. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing 
on the number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and 
the types of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating 
was determined for Literacy Express is in Appendix A6. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures 
and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. For the 
Literacy Express studies summarized here, no correction for clustering was needed. In the cases of Farver, Lonigan, and Eppe (2009) and Lonigan et al. 
(2005), a correction for multiple comparisons was needed, so the significance levels may differ from those reported in the original studies. 
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Effectiveness (continued) Development-Primary III (TOLD-P:3) Grammatic Understanding 

subtest. The analyses showed, and the WWC confirmed, that 
differences between Literacy Express and the control group cur- 
riculum are not statistically significant or substantively important 
on either of these measures. According to WWC criteria, this 
study shows no discernible effects on oral language. 

Print knowledge. Farver, Lonigan, and Eppe (2009) analyzed 
the effectiveness of Literacy Express on print knowledge using 
the Pre-CTOPPP Print Knowledge subtest. Their analyses 
showed, and the WWC confirmed, a statistically significant posi- 
tive effect of 0.64 when comparing the Literacy Express group 
and the control group. According to WWC criteria, this study 
shows a potentially positive effect on print knowledge. 

Lonigan et al. (2005) analyzed the effectiveness of Literacy 
Express on print knowledge using the Pre-CTOPPP Print Knowl- 
edge subtest. Their analyses showed, and the WWC confirmed, 
a statistically significant and substantively important positive 
effect of 0.32 when Literacy Express was compared to the 
control group curriculum. According to WWC criteria, this study 
shows a potentially positive effect on print knowledge. 

The PCER Consortium (2008) analyzed the effectiveness 
of Literacy Express on print knowledge using the Test of Early 
Reading Ability-Ill (TERA-3), the Woodcock-Johnson III (WJ-III) 
Letter-Word Identification subtest, and the WJ-III Spelling sub- 
test. The authors report, and the WWC confirms, that differences 
between Literacy Express and the control group curriculum are 
not statistically significant on any of these measures, although 
there is a substantively important effect of 0.30 on the WJ-III 
Letter-Word Identification subtest. According to WWC criteria, 
this study shows no discernible effects on print knowledge. 

Phonological processing. Farver, Lonigan, and Eppe (2009) 
analyzed the effect of Literacy Express on phonological process- 
ing using the Pre-CTCPPP Blending and Elision subtests. 

Their results showed, and the WWC confirmed, a statistically 
significant and substantively important positive effect of 0.54 
across the two measures (0.51 for Elision and 0.56 for Blending). 



According to WWC criteria, this study shows a potentially posi- 
tive effect on phonological processing. 

Lonigan et al. (2005) analyzed the effectiveness of Literacy 
Express on phonological processing using the Pre-CTCPPP 
Blending and Elision subtests. Their analyses showed, and 
the WWC confirmed, a statistically significant and substantively 
important positive effect of 0.26 across the two measures. 

This result was due primarily to a statistically significant and 
substantively important positive effect of 0.38 on the Elision 
subtest (the effect on the Blending subtest was neither statistically 
significant nor substantively important). According to WWC 
criteria, this study shows a potentially positive effect on phono- 
logical processing. 

The PCER Consortium (2008) analyzed the effectiveness 
ot Literacy Express on phonological processing using the 
Pre-CTCPPP Elision subtest. The authors report, and the WWC 
confirms, that the difference between the Literacy Express group 
and the control group is not statistically significant or substan- 
tively important on this measure. According to WWC criteria, this 
study shows no discernible effects on phonological processing. 

Cognition. Lonigan et al. (2005) analyzed the effectiveness of 
Literacy Express on cognition using three subtests from the Pre- 
CTCPPP— Non-Word Repetition, Word Span, and Rapid Cbject 
Naming. The authors report, and the WWC confirms, that the 
differences between the Literacy Express group and the control 
group were not statistically significant or substantively important 
for any of these three measures. According to WWC criteria, this 
study shows no discernible effects on cognition. 

Math. The PCER Consortium (2008) analyzed the effective- 
ness of Literacy Express on math using the WJ-III Applied Prob- 
lems subtest, the Child Math Assessment-Abbreviated (CMA-A), 
and the Shape Composition task. The authors report, and the 
WWC confirms, that differences between the Literacy Express 
group and the control group are not statistically significant or 
substantively important on any of these measures. According to 
WWC criteria, this study shows no discernible effects on math. 
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Effectiveness (continued) 



The WWC found 
Literacy Express to have 
positive effects on orai 
ianguage, print knowiedge, 
and phonoiogicai processing 
and no discernibie effects 
on cognition and math for 
preschooi chiidren 



References 



Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 



design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Proce- 
dures and Standards Handbook, Appendix F). The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition and the 
percentile rank of the average student in the comparison condi- 
tion. Unlike the rating of effectiveness, the improvement index is 
entirely based on the size of the effect, regardless of the statisti- 
cal significance of the effect, the study design, or the analysis. 
The improvement index can take on values between -50 and 
+50, with positive numbers denoting favorable results for the 
intervention group. 

Based on three studies, the average improvement index for 
Literacy Express on oral language is +12 percentile points, with a 
range of -2 to +23 percentile points across findings; the average 
improvement index on print knowledge Is +15 percentile points. 



with a range of +2 to +24 percentile points across findings; and 
the average improvement index on phonological processing Is 
+12 percentile points, with a range of +6 to +21 percentile points 
across findings. Based on one study, the average improvement 
index for Literacy Express on cognition is +1 percentile point, 
with a range of -5 to +5 percentile points across findings, and 
the average improvement index on math is 0 percentile points, 
with a range of -1 to +2 percentile points across findings. 

Summary 

The WWC reviewed four studies of Literacy Express for pre- 
school children. Three of these studies meet WWC evidence 
standards; the remaining study does not meet either WWC 
evidence standards or eligibility screens. Based on the three 
studies, the WWC found positive effects on oral language, 
print knowledge, and phonological processing and no discern- 
ible effects on cognition and math for preschool children. The 
conclusions presented in this report may change as new 
research emerges. 



Meet WWC evidence standards 

Farver, J. M., Lonigan, C. J., & Eppe, S. (2009). Effective early 
literacy skill development for young Spanish-speaking English 
language learners: An experimental study of two methods. 
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Lonigan, C. J., Farver, J. M., Clancy-Menchetti, J., & Phillips, B. M. 
(2005, April). Promoting the development of preschool chil- 
dren’s emergent literacy skills: A randomized evaluation 
of a literacy-focused curriculum and two professional devel- 
opment models. Paper presented at the biennial meeting of 
the Society for Research in Child Development, Atlanta, GA. 



Additional sources: 

Farver, J. A. M. (2005). Best practices in promoting literacy 
in young ethnic minority children: Lessons from the 
United States. Beijing, China: Soong Ching Ling Foundation 
and UNICEF. 
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tion of preliteracy skills. Early Education and Development, 
77(1), 91-114. 
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(2005, June). Promoting the development of preschool 
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(2008). Literacy Express and DLM Early Childhood Express 
supplemented with Open Court Reading Pre-K\ Florida State 
University. In Effects of preschool curriculum programs on 
school readiness (pp. 117-130). Washington, DC: U.S. Depart- 
ment of Education, Institute of Education Sciences, National 
Center for Education Research. 

Studies that fall outside the Early Childhood Education 

protocol or do not meet WWC evidence standards 

Lonigan, C. J. (2005, December). Impact of preschool literacy 
curricula: Results of a randomized evaluation in a public 



prekindergarten program. Paper presented at the annual 
meeting of the National Association for the Education 
of Young Children, Washington, DC. The study does not 
meet WWC evidence standards because it uses a quasi- 
experimental design in which the analytic intervention 
and comparison groups are not shown to be equivalent. 
Additional source: 

Lonigan, C. J. (2006, July). Impact of preschool literacy 
curricula: Results of a randomized evaluation in a public 
prekindergarten program. Paper presented at the 13th 
annual meeting of the Society for the Scientific Study 
of Reading, Vancouver, British Columbia, Canada. 



WWC Intervention Report Literacy Express 



July 2010 



Appendix 



Appendix A1.1 Study characteristics: Farver, Lonigan, & Eppe, 2009 



Characteristic 


Description 


Study citation 


Farver, J. M., Lonigan, C. J., & Eppe, S. (2009). Effective early literacy skill development for young Spanish-speaking English language learners: An experimental study ot two 
methods. Child Development, 80(3), 703-719. 


Participants 


Ninety-six Spanish-speaking English language learners in a Head Start program were randomly assigned, balancing for gender, to one of three conditions: (1) the High/Scope 
curriculum supplemented with small groups using Literacy Express in English only, (2) the High/Scope cumculum supplemented with small groups using Literacy Express 
beginning in Spanish and transitioning to English, and (3) the El igh/Scope cumculum only. Children were assigned to conditions within 10 classrooms. During the course otthe 
year, two children moved, resulting in a sample of 94 children (31 in English-only Literacy Express, 31 in the transitional Literacy Express, 32 in the control group). All children 
were born in the United States and lived in households in which Spanish was the primary language. Children receiving resource help for speech and language delays were not 
eligible for the study. The children in the sample were age 54.5 months, on average, and 46% were female. 


Setting 


The study was conducted in a Head Start program in inner-city Los Angeles, California. 


Intervention 


The intervention consisted of activities in dialogic reading, phonological awareness, and print knowledge. Dialogic reading activities included scaffolding techniques, such as 
asking “Wh-” and open-ended questions and using expansions and repetitions to encourage children to talk about the book. Phonological awareness involved word games that 
used picture puzzles to teach children that words were made of smaller sound units. Print knowledge activities taught children about the alphabet, including recognizing letters 
and their associated sounds. The intervention was delivered to small groups ot tour to five children in separate classrooms adjacent to the regular classrooms. The groups met 
for 20 minutes, tour times a week, trom mid-November to mid-May and were led by trained bilingual graduate students. In the Spanish-transition condition, instruction was 
in Spanish for the first eight weeks, followed by three to four weeks of transition. All lessons starting around week 14 were delivered in English. Children in the English-only 
condition received the tull 21 weeks of lessons in English.^ 


Comparison 


The comparison group received the High/Scope curriculum, which was typically offered in the center. 


Primary outcomes 
and measurement 


The primary outcome domains assessed were oral language, print knowledge, and phonological processing, all ot which were assessed with standardized measures. Oral 
language was assessed with the Receptive Vocabulary and Definitional Vocabulary subtests from the Preschool Comprehensive Test of Phonological and Print Processing 
(Pre-CTOPPP). Print knowledge was assessed with the Print Knowledge subtest from the Pre-CTOPPP. Phonological processing was assessed with the Blending and Elision 
subtests from the Pre-CTOPPP. Pretesting was done in fall of the preschool year, and posttesting was done in spring of the preschool year. Assessments were administered by 
trained research staff who were not involved in the delivery of the intervention and were blind to the children’s treatment status. Assessments were conducted with all children 
in English and Spanish; only the English assessments are used in the rating of the intervention. Outcomes for the Spanish assessments are included in Appendix 4.1 -A4.3, 
A4.5, A4.7, and A4.9. Eor a more detailed description of the outcome measures, see Appendices A2.1-A2.5. 


Staff/teacher training 


Eour bilingual graduate students were taught to deliver the intervention by one of the study's authors, who also supervised the intervention. No other information on training 
is provided. 



1. The study authors analyzed the English-only Literacy Express and transitional Literacy Express separately. The WWC uses the combined data to determine the rating of effectiveness for this 
WWC report, but provides the separate comparisons in Appendices A4.2-A4.9. 
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Appendix A1.2 Study characteristics: Lonigan, Farver, Ciancy-Menchetti, & Phiiiips, 2005 



Characteristic 


Description 


Study citation 


Lonigan, C. J., Farver, J. M., Ciancy-Menchetti, J., & Phillips, B. M. (2005, June). Promoting the development of preschool children’s emergent literacy skills: A randomized 
evaluation of a literacy-focused curriculum and two professional development models. Paper presented at the 12th annuai meeting of the Society for the Scientific Sfudy of 
Reading, Toronto, Ontario, Canada. 


Participants 


Eighteen preschoois from Tailahassee, Florida, and 30 preschools from Los Angeles, California, were randomly assigned to a /./teracy&press workshop group, a Literacy Express 
workshop plus mentoring group, or a control condition. This resulted in 15 preschools in each of the two Literacy Express groups and 18 preschools in the control group. The study 
began with 808 preschool children ranging in age from 36 to 69 months (mean age = 50.63 months). At posttest, 722 children remained in the sample and ranged in age from 
36 to 69 months (mean age = 50.71 months). The sample included 55.7% African-American children, 35.2% Latino/Flispanic children, 7.9% Caucasian children, and 1.1% 
children of other races and ethnicities. Forty-nine percent of the children were female; 52% of the children in the California sites and 1% of the children in the Florida sites were 
Spanish-speaking English language learners. All children were considered at risk for academic difficulties as determined by pretest scores on a measure of cognitive performance. 


Setting 


The study took place in 18 preschools in Tallahassee, Florida, and 30 preschools in Los Angeles, California. The majority of the preschools (77%) were Flead Start programs. 


Intervention 


Preschools in the intervention group participated in a Literacy Express plus professional development via workshops group (“workshop group”) or a Literacy Express plus 
professional development via workshops and mentoring group (“mentoring group”). Literacy Express is a preschool literacy-focused curriculum that is intended to promote 
children’s emergent literacy skills. The curriculum used in this study (an earlier version) was structured around 11 thematic units (with games and activities in each unit) that 
were sequenced in order of complexity. In the earlier and the current versions of this curriculum, each unit includes children’s books that address theme-relevant vocabulary 
for small- and large-group reading activities. Each thematic unit includes small-group activities that provide children with the opportunity to attend to and practice skills 
related to oral language, phonological sensitivity, and print awareness and to receive individual feedback. Small-group activities are conducted three or four times a week. 

The curriculum provides guidance to teachers on grouping children who are progressing at similar rates. Large-group and extension activities provide opportunities for children 
to use new skills. The workshop group teachers participated in two-day workshops at the beginning of the school year and three half-day workshops during the school year. 
The mentoring group teachers participated in the same workshops and received regular classroom visits by a trained project mentor.^ 


Comparison 


Classrooms in the business-as-usual comparison group participated in the preschool’s standard curriculum, which in most cases was High/Scope or Creative Curriculurrf’ . 


Primary outcomes 
and measurement 


The primary outcome domains assessed were children’s oral language, print knowledge, phonological processing, and cognition, all of which were assessed with standardized 
measures. Oral language was assessed with the Expressive Communication subtest from the Preschool Language Scales-IV (PLS-IV). Print knowledge was assessed with 
the Print Knowledge subtest from the Preschool Comprehensive Test of Phonological and Print Processing (Pre-CTOPPP). Phonological processing was assessed with the 
Blending and Elision subtests from the Pre-CTOPPP. Cognition was assessed with three subtests from the Pre-CTOPPP: Non-Word Repetition, Word Span, and Rapid Object 
Naming. Pretesting was done in fall of the preschool year, and posttesting was done in spring of the preschool year. Trained research staff administered all assessments in a 
quasi-random order at pretest and posttest. All assessments were conducted with all children in English. Research staff also observed the study classrooms for three hours 
twice a year to determine implementation fidelity and to administer two general measures of classroom language and literacy, but these measures are not discussed further in 
this WWC intervention report. Eor a more detailed description of the outcome measures, see Appendices A2.1-A2.5. 


Staff/teacher training 


The research staff provided all materials and training for the Literacy Express intervention groups. Classroom teachers and aides attended a two-day curriculum-specific pro- 
fessional development workshop at the start of the school year, as well as three half-day curriculum-specific professional development workshops throughout the school year. 
In all workshops, staff participated in both teacher-directed and hands-on components. Classroom staff in the mentoring group received regular classroom visits throughout 
the school year from trained project teacher-mentors in addition to the professional development activities. 



1 . The study authors combined the two intervention groups to examine the effectiveness of Literacy Express, and the WWC uses the combined data to determine the rating of effectiveness for this WWC 
intervention report. The study authors provide comparisons between the Literacy Express mentoring and workshop groups and the controi group, between Engiish speakers in the Caiifornia sampie and 
the control group, and between Spanish speakers in the California sample and the control group. The WWC includes the results from these analyses in Appendices A4.1-A4.4. The study authors also 
provide comparisons between the Literacy Express mentoring group and the Literacy Express workshop group. The WWC includes the results from these analyses in Appendices A4.5-A4.8. 
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Appendix A1.3 Study characteristics: Preschooi Curricuium Evaiuation Research (PCER) Consortium, 2008 



Characteristic 


Description 


Study citation 


Preschool Curriculum Evaluation Research (PCER) Consortium. (2008). Literacy Express anti DLM Early Childhood Express supplemented with Open Court Reading Pre-K: 
Florida State University. In Effects of preschool curriculum projects on school readiness (pp. 117-130). Washington, DC: U.S. Department of Education, Institute ot Education 
Sciences, National Center for Education Research. 


Participants 


This study, conducted during the 2003/04 and 2004/05 school years, included three intervention groups: Literacy Express, DLM Early Childhood Express supplemented with 
Open Court Reading, and a control group. Eighteen schools were initially recruited to participate in the study. Sixteen of these were assigned a letter grade ot A, B, C, or D as 
part of Florida's school grading system (2 schools were not part of the grading system and were dropped from the study). School grades were used as a blocking variable, and 
schools within each grade were ranked on average number of years of teaching experience. One additional school joined the study late (for a total of 17 participating schools). 
Schools were then grouped into triplets and randomized into three conditions: Literacy Express (6 schools), DLM(5 schools), and control (6 schools). Although schools were 
randomized into three groups, this review is restricted to a comparison of Literacy Express mtti the control group.^ School is the unit of assignment; if a school had multiple 
preschool classrooms, all of those classrooms were assigned to the same intervention. The study as reviewed included 6 Literacy Express and 6 control schools and 1 0 
Literacy Express and 9 control classrooms and began with a total of 1 96 children (99 Literacy Express and 97 control). The parental consent rate was 95% tor the combined 
treatment group and 93% for the control group. At baseline, children in the study averaged age 4.6; 54% were male; and 6% were Hispanic, 30% were Caucasian, and 59% 
were African American. The analysis sample included between 177 and 188 children, depending on the outcome measure. There was no attrition ot schools. Depending on 
the outcome, child-level attrition ranged from 6% to 10% for Literacy Express and from 2% to 9% for the control group. 


Setting 


The Literacy Express study was conducted with children from 12 schools and 19 classrooms selected from public prekindergarten programs in Florida. 


Intervention 


Literacy Express is a preschool literacy-focused curriculum that is intended to promote children’s emergent literacy skills. The version of the curriculum used in this study was 
structured around 11 thematic units (with games and activities in each unit). The version used in this study and the current version of the curriculum sequence the units in 
order of complexity. Each unit includes children’s books that address theme-relevant vocabulary for small- and large-group reading activities. Each thematic unit includes 
small-group activities that provide children with the opportunity to attend to and practice skills related to oral language, phonological sensitivity, and print awareness and to 
receive individual feedback. Small-group activities are conducted three or four times a week. The curriculum provides guidance to teachers on grouping children who are 
progressing at similar rates. Large-group and extension activities provide opportunities for children to use new skills. Fidelity observations were conducted in treatment and 
control classrooms during February 2004 and April/May 2004 using the Early Language and Literacy Classroom Observation (ELLCO) tool and the Center for Improving the 
Readiness of Children for Learning and Education (CIRCLE) observation tools. Observations lasted 2.5 to 3 hours in each classroom. Researchers used site-specific implemen- 
tation and fidelity data to rate each treatment classroom on the global fidelity measure as High (3.0), Medium (2.0), Low (1 .0), or Not at All (0.0). Researchers also provided a 
global rating for the control group. Literacy Express was rated in the High-Medium range (2.5) on the global implementation fidelity measure, whereas the control group was 
rated at the Medium level (2.0). 


Comparison 


Teachers of control group classrooms were trained to use the High/Scope curriculum. Training provided to teachers in control classrooms included a week-long summer insti- 
tute conducted by High/Scope trainers prior to the start of the project, additional training sessions throughout the school year conducted by High/Scope and district personnel, 
and classroom visits by the High/Scope trainer. 



(continued) 
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Appendix A1.3 Study characteristics: Preschooi Curricuium Evaiuation Research (PCER) Consortium, 2008 (continued) 



Characteristic 


Description 


Primary outcomes 
and measurement 


The primary outcome domains assessed were children’s oral language, print knowledge, phonological processing, and math, all of which were assessed with standardized 
measures. Oral language was assessed with the Peabody Picture Vocabulary Test III (PPVT-III) and the Grammatic Understanding subtest from the Test of Language 
Development-Primary III (T0LD-P:3). Print knowledge was assessed with the Test of Early Reading Ability-Ill (TERA-3) and the Woodcock- Johnson III (WJ-III) Letter-Word 
Identification and Spelling subtests. Phonological processing was assessed with the Elision subtest from the Preschool Comprehensive Test of Phonological and Print Process- 
ing (Pre-CT0PPP).2 Math was assessed with the WJ-III Applied Problems subtest, the Composite Score from the Child Math Assessment-Abbreviated (CMA-A), and the 
Building Blocks Shape Composition test. Pretesting was done in fall of the preschool year, and posttesting was done in spring of the preschool year.^ Trained research staff 
administered all assessments, which were conducted with all children in English. Research staff also observed the study classrooms for three hours twice a year using the 
ELLCO and CIRCLE observation measures, but these measures are not discussed further in this WWC intervention report. For a more detailed description of these outcome 
measures, see Appendices A2.1-A2.5. 


Staff/teacher training 


Teachers received curriculum training from the Florida research team for four days in July 2003, prior to the start of the 2003/04 school year. The first two days of the training 
were spent in a workshop setting, and the other two days were used for team planning. The workshop training session familiarized teachers and their aides with the curriculum 
materials and provided hands-on experience in leading curricular activities. Videotaped training was made available for teachers who could not attend in person. Teachers and 
aides attended a two-hour training session every other month during the school year. 



1 . For the rating of effectiveness in this WWC intervention report, the WWC inciudes oniy the resuits comparing Literacy Express with the controi group. The WWC does not inciude the DLM 
Express pius Open Court Pre-K versus controi resuits in a separate WWC intervention report because the effects of DLM Express and Open Court Pre-K on chiidren’s outcomes cannot be 
disentangied. The WWC does not inciude the head-to-head comparison of Literacy Express and DLM Express pius Open Court Pre-K, but interested readers can examine that comparison 
using the data provided in the originai articie. 

2. The CTOPP Elision test was used in the kindergarten follow-up reported in Appendix A4.14. 

3. The study authors also provide comparisons of the Literacy Express group with the control group based on follow-up data collected in spring of the kindergarten year. The WWC includes the 
results from these analyses in Appendices A4.9-A4.12. 
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Appendix A2.1 Outcome measures for the oral language domain 



Outcome measure 


Description 


Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTOPPP) — Receptive Vocabulary subtest 


A subtest from a standardized measure that assesses receptive vocabulary by asking a child to point to one of four pictures that 
best represents a word spoken by the assessor (as cited in Farver, Lonigan, & Eppe, 2009). 


Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTOPPP) — Definitional Vocabulary subtest 


A subtest from a standardized measure that assesses children’s single-word oral and expressive vocabulary. The child is asked 
to name what is shown in a picture and describe one important feature (as cited in Farver, Lonigan, & Eppe, 2009). 


Peabody Picture Vocabulary Test-Ill (PPVT-III) 


A standardized measure of children’s receptive vocabulary in which children show understanding of a spoken word by pointing 
to a picture that best represents the meaning (as cited in PCER Consortium, 2008). 


Test of Language Development-Primary III (T0LD-P:3) — 
Grammatic Understanding subtest 


A standardized measure of children’s ability to comprehend the meaning of sentences by selecting pictures that most accurately 
represent the sentence (as cited in PCER Consortium, 2008). 


Preschool Language Scales-IV (PLS-IV) — 
Expressive Communication subtest 


A subtest from a standardized measure that assesses children’s expressive communication skills in multiple areas of language development 
(vocal development, social communication, semantics, structure, and integrative thinking) (as cited in Lonigan et al., 2005). 



Appendix A2.2 Outcome measures for the print knowledge domain 



Outcome measure 


Description 


Test of Early Reading Ability-Ill (TERA-3) — Total Score 


A Standardized measure of children’s developing reading skills with three subtests: Alphabet, Conventions, and Meaning (as cited in 
PCER Consortium, 2008).'' 


Woodcock-Johnson III (WJ-III) Letter- 
Word Identification subtest 


A standardized measure of identification of letters and reading of words (as cited in PCER Consortium, 2008). 


Woodcock-Johnson III (WJ-III) Spelling subtest 


A standardized measure that assesses children’s prewriting skills, such as drawing lines, tracing, and writing letters (as cited in PCER 
Consortium, 2008). 


Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTDPPP) — Print Knowledge subtest 


A subtest from a standardized measure that assesses children’s early print concepts, alphabet recognition, letter-name knowledge, 
and letter-sound knowledge (as cited in Lonigan et al., 2005 and Farver, Lonigan, & Eppe, 2009). 



1. By name, this measure sounds as if it shouid be captured under the eariy reading and writing domain; however, the description of the measure identifies constructs that are pertinent to print 
knowiedge, such as knowing the aiphabet, understanding print conventions, and environmentai print. 
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Appendix A2.3 Outcome measures for the phonological processing domain 



Outcome measure 


Description 


Preschool Comprehensive Test of Phonological and 
Print Processing (Pre-CTOPPP) — Elision subtest 


A measure of children’s ability to identify and manipulate sounds in spoken words, using word prompts and picture plates tor the tirst 
nine items and word prompts only for later items (as cited in PCER Consortium, 2008). 


Preschool Comprehensive Test of Phonological and 
Print Processing (Pre-CTOPPP) — Blending subtest 


A subtest from a standardized measure that requires children to combine words, syllables, or phonemes to make a new word (as cited in 
Farver, Lonigan, & Eppe, 2009). 


Comprehensive Test of Phonological 
Processing (CTOPP) — Elision subtest 


The CTOPP Elision subtest assesses phonological awareness and is similar to the Pre-CTOPPP Elision subtest but does not include 
pictures in the administration format (as cited in PCER Consortium, 2008). 



Appendix A2.4 Outcome measures for the cognition domain 



Outcome measure 


Description 


Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTOPPP) — Non-Word Repetition subtest 


A subtest from a standardized measure that assesses children’s auditory short-term memory by having children repeat non-words built 
from English phonology that grow increasingly longer throughout the assessment (as cited in Lonigan et al., 2005). 


Preschool Comprehensive Test of Phonological and 
Print Processing (Pre-CTOPPP) — Word Span subtest 


A subtest from a standardized measure that assesses children’s auditory short-term memory by having children repeat one- to seven- 
word lists of common words (as cited in Lonigan et al., 2005). 


Preschool Comprehensive Test of Phonological and Print 
Processing (Pre-CTOPPP) — Rapid Object Naming subtest 


A subtest from a standardized measure that assesses children’s lexical access by measuring the speed with which children can name 
pictures of five common objects that are arranged randomly within rows (as cited in Lonigan et al., 2005). To make effect size estimates 
consistent across measures, the WWC reversed the direction of the effect so that a higher score reflected a better outcome. 



Appendix A2.5 Outcome measures for the math domain 



Outcome measure 


Description 


Woodcock-Johnson III (WJ-III) Applied Problems subtest 


A standardized measure of children’s ability to solve numerical and spatial problems, presented verbally with accompanying pictures of 
objects (as cited in PCER Consortium, 2008). 


Child Math Assessment-Abbreviated (CMA-A) 
Composite Score 


The average of four subscales: (1) solving addition and subtraction problems using visible objects, (2) constructing a set of objects equal 
in number to a given set, (3) recognizing shapes, and (4) copying a pattern using objects that vary in color and identity from the model 
pattern (as cited in PCER Consortium, 2008). 


Building Blocks, Shape Composition task 


Modified for PCER from the Building Blocks assessment tools. Children use blocks to fill in a puzzle and are assessed on whether they 
fill the puzzle without gaps or hangovers (as cited in PCER Consortium, 2008). 
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Appendix A3.1 Summary of study findings inciuded in the rating for the orai ianguage domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Receptive 
Vocabulary subtest 


Preschoolers 


10/94 


Farver, Lonigan, & Eppe, 2009® 

31.21 28,33 

(4,99) (5.63) 


2.88 


0,55 


Statistically 

significant 


+2^ 


Pre-CTOPPP Definitional 
Vocabulary subtest 


Preschoolers 


10/94 


49.87 

(12,66) 


41.23 

(16.85) 


8.64 


0,60 


Statistically 

significant 


+23 


Average for oral language (Farver, Lonigan, & Eppe, 2009)® 








0.57 


Statistically 

significant 


+22 








Lonigan et al., 2005® 










PLS-IV Expressive 


Preschoolers 


48/722 


53.35 


50.66 


2.69 


0,30 


Statistically 


-Hi 2 


Communication subtest 






(8.78) 


(9,71) 






significant 




Average for oral language (Lonigan et al., 2005)® 










0.30 


Statistically 


+12 
















significant 










PCER Consortium, 2008® 










PPVT-III 


Preschoolers 


12/186 


90.30 


87.31 


2.99 


0,17 


ns 


+7 








(15.43) 


(13,77) 










T0LD-P:3 Grammatic 


Preschoolers 


12/181 


8,11 


8.33 


-0,22 


-0,04 


ns 


-2 


Understanding subtest 






(2,62) 


(2.71) 










Average for oral language (PCER Consortium, 2008)® 








0.06 


ns 


+2 


Domain average for oral language across all studies® 








0.31 


na 


+12 



ns = not statistically significant 
na = not applicable 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 
PLS-IV = Preschool Language Scale-IV 
PPVT-III = Peabody Picture Vocabulary Test-Ill 
T0LD-P:3 = Test of Language Development-Primary III 
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Appendix A3.1 Summary of study findings inciuded in the rating for the orai ianguage domain (continued) 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices for the orai ianguage domain. The findings of Farver, Lonigan, and Eppe (2009) 
presented in this tabie combine the Literacy Express Engiish-oniy group with the Literacy Express transitionai group and compare chiidren in this combined treatment group with chiidren in the 
comparison group. Findings for each of these groups are presented separateiy in Appendices A4.1 and A4.5. Subgroup findings from Lonigan et ai. (2005) are not inciuded in these ratings but 
are reported in Appendices A4.1 and A4.6. Foiiow-up findings from PCER Consortium (2008) are not inciuded in these ratings but are reported in Appendix A4.12. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants 
had more simiiar outcomes. 

3. in the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronoiogicai age and pretest, in the case of PCER Consortium (2008), each intervention group mean is caicuiated 
as the sum of the unadjusted controi mean and the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an expianation of the effect size caicuiation, see WWC Procedures and Standards Fiandbook, Appendix B. in the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

7. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on vaiues between -50 and +50, with positive numbers denoting favorabie resuits for the intervention group. 

8. The ievei of statisticai significance was reported by the study authors or, when necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie compari- 
sons. For the formuias the WWC used to caicuiate the statisticai significance, see WWC Procedures and Standards Fiandbook, Appendix C for ciustering and WWC Procedures and Standards 
Fiandbook, Appendix D for muitipie comparisons, in the case of Farver, Lonigan, and Eppe (2009), a correction for ciustering was not needed; however, a correction for muitipie comparisons was 
needed, so the significance ieveis may differ from the originai study, in the case of Lonigan et ai. (2005), no corrections for muitipie comparisons were needed and no correction for ciustering 
was needed because the anaiysis corrected for ciustering by using hierarchicai iinear modeis (FiLM). in the case of PCER Consortium (2008), no corrections for ciustering or muitipie compari- 
sons were needed because the anaiysis corrected for ciustering by using FiLM, and no impacts were statisticaiiy significant. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simpie averages rounded to two decimai piaces. The average improvement indices are caicuiated 
from the average effect sizes. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the print knowiedge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Print 
Knowledge subtest 


Preschoolers 


10/94 


Farver, Lonigan, & Eppe, 2009® 

22,01 16.61 

(8,47) (7,96) 


5.40 


0,64 


Statistically 

significant 


-h24 


Average for print knowledge (Farver, Lonigan, & Eppe, 2009)® 








0.64 


Statistically 

significant 


+24 








Lonigan et al., 2005® 










Pre-CTOPPP Print 


Preschoolers 


48/722 


17.55 


14.70 


2.85 


0.32 


Statistically 


-Hi 3 


Knowledge subtest 






(9.30) 


(7.83) 






significant 




Average for print knowledge (Lonigan et al., 2005)® 


1 








0.32 


Statistically 


+13 
















significant 










PCER Consortium, 2008® 










TERA-3 


Preschoolers 


12/183 


82.36 


81.10 


1.26 


0,17 


ns 


+7 








(12,45) 


(11.00) 










WJ-III Letter-Word 


Preschoolers 


12/183 


103.30 


95.60 


7.70 


0,30 


ns 


-Hi 2 


Identification subtest 






(14.10) 


(12,38) 










WJ-III Spelling subtest 


Preschoolers 


12/182 


89.03 


87.67 


1.36 


0.05 


ns 


-h2 








(14,34) 


(11.75) 










Average for print knowledge (PCER Consortium, 2008)® 








0.17 


ns 


+7 


Domain average for print knowledge across all studies® 








0.38 


na 


+15 



ns = not statistically significant 
na = not applicable 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 
TERA-3 = Test of Early Reading Ability-Ill 
WJ-III = Woodcock-Johnson III 
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Appendix A3.2 Summary of study findings inciuded in the rating for the print knowiedge domain (continued) 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the print knowiedge domain. The findings of Farver, Lonigan, and Eppe (2009) 
presented in this tabie combine the Literacy Express Engiish-oniy group and the Literacy Express transitionai group and compare chiidren in this combined group with chiidren in the comparison 
group. Findings for each of these groups are presented separateiy in Appendices A4.2 and A4.7. Subgroup findings from Lonigan et ai. (2005) are not inciuded in these ratings but are reported in 
Appendix A4.2 and A4.8. Foilow-up findings from PCER Consortium (2008) are not inciuded in these ratings but are reported in Appendix A4.13. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants 
had more simiiar outcomes. 

3. in the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronoiogicai age and pretest, in the case of PCER Consortium (2008), each intervention group mean is caicuiated 
as the unadjusted controi mean pius the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an expianation of the effect size caicuiation, see WWC Procedures and Standards Fiandbook, Appendix B. in the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

7. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on vaiues between -50 and +50, with positive numbers denoting favorabie resuits for the intervention group. 

8. The ievei of statisticai significance was reported by the study authors or, when necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie compari- 
sons. For the formuias the WWC used to caicuiate the statisticai significance, see WWC Procedures and Standards Fiandbook, Appendix C for ciustering and WWC Procedures and Standards 
Fiandbook, Appendix D for muitipie comparisons, in the case of Farver, Lonigan, and Eppe (2009), a correction for ciustering was not needed; however, a correction for muitipie comparisons 
was needed, so the significance ieveis may differ from the originai study, in the case of Lonigan et ai. (2005), no correction for muitipie comparisons was needed, and no correction for ciustering 
was needed because the anaiysis corrected for ciustering by using FILM. In the case of PCER Consortium (2008), no corrections for clustering or multiple comparisons were needed because the 
analysis corrected for clustering by using FILM, and no impacts were statistically significant. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.3 Summary of study findings inciuded in the rating for the phonoiogicai processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Blending subtest 


Preschoolers 


10/94 


Farver, Lonigan, & Eppe, 2009® 

14.37 12.69 

(3.16) (3.51) 


1.68 


0.51 


Statistically 

significant 


-Hi 9 


Pre-CTOPPP Elision subtest 


Preschoolers 


10/94 


8.00 

(3.35) 


6.37 

(1.51) 


1.63 


0.56 


Statistically 

significant 


-h21 


Average for phonological processing (Farver, Lonigan, & Eppe, 2009)® 








0.54 


Statistically 

significant 


+20 








Lonigan et al., 2005® 










Pre-CTOPPP Blending subtest 


Preschoolers 


48/722 


14.15 


13.47 


0.68 


0.15 


ns 


-h6 








(4.47) 


(4.47) 










Pre-CTOPPP Elision subtest 


Preschoolers 


48/722 


8.86 


7.43 


1.43 


0.38 


Statistically 


-Hi 5 








(3.93) 


(3.54) 






significant 




Average for phonological processing (Lonigan et al., 2005)® 








0.26 


Statistically 


+10 
















significant 










PCER Consortium, 2008® 










Pre-CTOPPP Elision subtest 


Preschoolers 


12/188 


9.42 


8.79 


0.63 


0.14 


ns 


-h6 








(4.54) 


(4.37) 










Average for phonological processing (PCER Consortium, 2008)® 








0.14 


ns 


+6 


Domain average for phonological processing across all studies® 








0.31 


na 


+12 



ns = not statistically significant 
na = not applicable 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 
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Appendix A3.3 Summary of study findings inciuded in the rating for the phonoiogicai processing domain (continued) 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices for the phonoiogicai processing domain. The findings of Farver, Lonigan, and Eppe 
(2009) presented in this tabie combine the Literacy Express Engiish-oniy group and the Literacy Express transitionai group and compare chiidren in this combined group with chiidren in the 
comparison group. Findings for each of these groups are presented separateiy in Appendices A4.3 and A4.9. Subgroup findings from Lonigan et ai. (2005) are not inciuded in these ratings but 
are reported in Appendices A4.3 and A4.10. Foiiow-up findings from PCER Consortium (2008) are not inciuded in these ratings but are reported in Appendix A4.14. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants 
had more simiiar outcomes. 

3. in the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronoiogicai age and pretest, in the case of PCER Consortium (2008), each intervention group mean is caicuiated 
as the unadjusted controi mean pius the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an expianation of the effect size caicuiation, see WWC Procedures and Standards Handbook, Appendix B. in the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

7. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on vaiues between -50 and +50, with positive numbers denoting favorabie resuits for the intervention group. 

8. The ievei of statisticai significance was reported by the study authors or, when necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie compari- 
sons. For the formuias the WWC used to caicuiate the statisticai significance, see WWC Procedures and Standards Handbook, Appendix C for ciustering and WWC Procedures and Standards 
Handbook, Appendix D for muitipie comparisons, in the case of Farver, Lonigan, and Eppe (2009), a correction for ciustering was not needed; however, a correction for muitipie comparisons was 
needed, so the significance ieveis may differ from the originai study, in the case of Lonigan et ai. (2005), a correction for muitipie comparisons was needed, so the significance ieveis may differ 
from those reported in the originai study, in the case of PCER Consortium (2008), no corrections for ciustering or muitipie comparisons were needed because the anaiysis corrected for ciuster- 
ing by using HLM, and no impacts were statisticaiiy significant. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simpie averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.4 Summary of study findings inciuded in the rating for the cognition domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

group group 


Mean 

difference^ 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Non-Word 
Repetition subtest 


Preschooiers 


48/722 


9.07 

(4.16) 


Lonigan et al., 2005^ 

9.55 

(4,36) 


-0.48 


-0,11 


ns 


-5 


Pre-CTOPPP Word Span subtest 


Preschooiers 


48/722 


8.86 

(2.56) 


8.54 

(2.54) 


0.32 


0.13 


ns 


+5 


Pre-CTOPPP Rapid Object 
Naming subtest 


Preschoolers 


48/722 


48,21 

(17,12) 


49.57 

(16.93) 


1.36« 


0,08 


ns 


+3 


Domain average for cognition^ 












0.03 


na 


+1 



ns = not statistically significant 
na = not applicable 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices for the cognition domain. Subgroup findings from Lonigan et al. (2005) are not 
included in these ratings but are reported in Appendix A4.5 and A4.11. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no correction for clustering was needed and no correction for multiple comparisons was needed because 
no impacts were statistically significant. 

8. For this outcome, the mean difference was calculated so that a positive effect was found when intervention group children took less time than comparison group children to complete the task 
(comparison group mean minus the intervention group mean). 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. 

The domain improvement index is calculated from the average effect size. 
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Appendix A3.5 Summary of study findings inciuded in the rating for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008^ 



WJ-III Applied Problems subtest 


Preschoolers 


12/177 


89.12 

(15,37) 


87,86 

(13.77) 


1,26 


0,05 


ns 


-f2 


CMA-A Composite 


Preschoolers 


12/185 


0.51 

(0.24) 


0.52 

(0.21) 


-0.01 


-0,02 


ns 


-1 


Shape Composition 


Preschoolers 


12/185 


1.54 

(1.00) 


1.55 

(0.94) 


-0.01 


-0.01 


ns 


0 


Oomain average for math® 












0.01 


na 


0 



ns = not statistically significant 

na = not applicable 

WJ-III = Woodcock-Johnson III 



CMA-A = Child Math Assessment-Abbreviated 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math domain. Follow-up findings from PCER Consortium (2008) are not 
included in these ratings but are reported in Appendix A4.15. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. For PCER Consortium (2008), each intervention group mean is calculated as the unadjusted control mean plus the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards 
Flandbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections for clustering or multiple comparisons were needed because the analysis corrected for 
clustering by using FILM, and no impacts were statistically significant. 

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. 

The domain improvement index is calculated from the average effect size. 
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Appendix A4.1 Summary of subgroup findings for the orai ianguage domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Receptive 
Vocabulary subtest (English) 


Preschoolers 


10/63 


Farver, Lonigan, & Eppe, 2009 (English-only group)® 

30.62 28.33 2.29 

(5.85) (5.63) 


0.39 


ns 


-el 5 


Pre-CTOPPP Definitional 
Vocabulary subtest (English) 


Preschoolers 


10/63 


47.45 

(12.96) 


41.23 

(16.85) 


6.22 


0.41 


ns 


-Hie 


Pre-CTOPPP Receptive 
Vocabulary subtest (Spanish) 


Preschoolers 


10/63 


24.58 

(4.07) 


23.79 

(4.03) 


0.79 


0.19 


ns 


-h8 


Pre-CTOPPP Definitional 
Vocabulary subtest (Spanish) 


Preschoolers 


10/63 


25.90 

(19.30) 


25.74 

(15.97) 


0.16 


0.01 


ns 


0 








Farver, Lonigan, & Eppe, 2009 (transitional group)® 








Pre-CTOPPP Receptive 


Preschoolers 


10/63 


31.79 


28.33 


3.46 


0.70 


Statistically 


-h26 


Vocabulary subtest (English) 






(3.95) 


(5.63) 






significant 




Pre-CTOPPP Definitional 


Preschoolers 


10/63 


52.28 


41.23 


11.05 


0.74 


Statistically 


-h27 


Vocabulary subtest (English) 






(12.07) 


(16.85) 






significant 




Pre-CTOPPP Receptive 


Preschoolers 


10/63 


27.03 


23.79 


3.24 


0.65 


Statistically 


-h24 


Vocabulary subtest (Spanish) 






(5.74) 


(4.03) 






significant 




Pre-CTOPPP Definitional 


Preschoolers 


10/63 


32.66 


25.74 


6.92 


0.40 


ns 


-Hi 5 


Vocabulary subtest (Spanish) 






(18.30) 


(15.97) 
















Lonigan et al., 2005 (workshop group)® 










PLS-IV Expressive 


Preschoolers 


33/442 


52.63 


50.66 


1.97 


0.20 


ns 


-h8 


Communication subtest 






(9.68) 


(9.71) 
















Lonigan et al., 2005 (mentoring group)® 










PLS-IV Expressive 


Preschoolers 


33/516 


54.04 


50.66 


3.38 


0.38 


Statistically 


-Hi 5 


Communication subtest 






(7.97) 


(9.71) 






significant 








Lonigan et al., 2005 (English speakers in the California sample)® 








PLS-IV Expressive 


Preschoolers 


30/198** 


55.12 


51.02 


4.10 


0.59 


Statistically 


-h22 


Communication subtest 






(6.40) 


(8.01) 






significant 





(continued) 
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Appendix A4.1 Summary of subgroup findings for the orai ianguage domain (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PLS-IV Expressive 


Preschoolers 


Lonigan et al., 2005 (Spanish speakers in the California sample)® 

30/231® 46,62 44.04 2.58 


0,25 


ns 


+10 


Communication subtest 




(10.11) (10,76) 









ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 
PLS-IV = Preschool Language Scales-IV 



1 . This appendix presents subgroup findings for measures that fall in the oral language domain. Total group scores were used for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronological age and pretest. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Farver, Lonigan, and Eppe (2009), a correction 
for clustering was not needed; however, a correction for multiple comparisons was needed, so the significance levels may differ from the original study. In the case of Lonigan et al. (2005), no 
correction for multiple comparisons was needed and no correction for clustering was needed because the analysis used HLM. 

9. The number of preschools in each group at assignment and the number of children in each group at posttest was provided by the study authors upon WWC request. 
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Appendix A4.2 Summary of subgroup findings for the print knowiedge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
(Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Print 
Knowledge subtest (English) 


Preschoolers 


Farver, Lonigan, & Eppe, 2009 (English-only group)® 

10/63 20.11 16.61 3.50 

(9.01) (7.96) 


0.41 


ns 


-Hie 


Pre-CTOPPP Print 
Knowledge subtest (Spanish) 


Preschoolers 

1 


10/63 


13.14 

(6.62) 


12.83 

(6.28) 


0.31 


0.05 


ns 


-h2 






Farver, Lonigan, & Eppe, 2009 (transitional group)® 








Pre-CTOPPP Print 


Preschoolers 


10/63 


23.90 


16.61 


7.29 


0.93 


Statistically 


-h32 


Knowledge subtest (English) 






(7.56) 


(7.96) 






significant 




Pre-CTOPPP Print 


Preschoolers 


10/63 


16.54 


12.83 


3.71 


0.48 


Statistically 


-Hi 8 


Knowledge subtest (Spanish) 




(8.90) 


(6.28) 






significant 










Lonigan et al., 2005 (workshop group)® 








Pre-CTOPPP Print 


Preschoolers 


33/442 


16.19 


14.72 


1.47 


0.17 


ns 


+7 


Knowledge subtest 






(9.21) 


(7.83) 










Lonigan et al., 2005 (mentoring group)® 


Pre-CTOPPP Print 


Preschoolers 


33/516 


18.77 


14.72 


4.05 


0.47 


Statistically 


-Hi 8 


Knowledge subtest 






(9.16) 


(7.83) 






significant 








Lonigan et al., 2005 (English speakers in the California sample)® 








Pre-CTOPPP Print 


Preschoolers 


30/1989 


17.40 


15.32 


2.08 


0.23 


ns 


-h9 


Knowledge subtest 






(9.51) 


(8.30) 














Lonigan et al., 2005 (Spanish speakers in the California sample)® 








Pre-CTOPPP Print 


Preschoolers 


30/2319 


15.32 


11.29 


4.03 


0.52 


Statistically 


-h20 


Knowledge subtest 






(8.73) 


(5.21) 






significant 





ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



(continued) 
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Appendix A4.2 Summary of subgroup findings for the print knowiedge domain (continued) 

1. This appendix presents subgroup findings for measures that faii in the print knowiedge domain. Totai group scores were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants 
had more simiiar outcomes. 

3. in the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronoiogical age and pretest. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an expianation of the effect size caicuiation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

7. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on vaiues between -50 and +50, with positive numbers denoting resuits favorabie to the intervention group. 

8. The ievei of statisticai significance was reported by the study authors or, when necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois (corrections for muitipie 
comparisons were not done for findings not inciuded in the overaii intervention rating). For the formuias the WWC used to calcuiate the statisticai significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for ciustering and WWC Procedures and Standards Handbook, Appendix D for muitipie comparisons. In the case of Farver, Lonigan, and Eppe (2009), a correction 
for ciustering was not needed; however, a correction for muitipie comparisons was needed, so the significance ieveis may differ from the originai study. In the case of Lonigan et ai. (2005), no 
correction for muitipie comparisons was needed, and no correction for ciustering was needed because the anaiysis used HLM. 

9. The number of preschoois in each group at assignment and the number of chiidren in each group at posttest was provided by the study authors upon WWC request. 
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Appendix A4.3 Summary of subgroup findings for the phonoiogicai processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference’' 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 









Farver, Lonigan, & Eppe, 2009 (English-only group)® 








Pre-CTOPPP Blending subtest 
(English) 


Preschoolers 


10/63 


14.31 

(3.33) 


12.69 

(3.51) 


1.62 


0.47 


ns 


-Hi 8 


Pre-CTOPPP Elision subtest 
(English) 


Preschoolers 


10/63 


7.96 

(3.24) 


6.37 

(1.51) 


1.59 


0.62 


Statistically 

significant 


-h23 


Pre-CTOPPP Blending subtest 
(Spanish) 


Preschoolers 


10/63 


11.13 

(2.49) 


10.59 

(3.02) 


0.54 


0.19 


ns 


-h8 


Pre-CTOPPP Elision subtest 
(Spanish) 


Preschoolers 


10/63 


5.94 

(1.75) 


5.52 

(1.32) 


0.42 


0.27 


ns 


-hII 








Farver, Lonigan, & Eppe, 2009 (transitional group)® 








Pre-CTOPPP Blending subtest 
(English) 


Preschoolers 


10/63 


14.43 

(3.04) 


12.69 

(3.51) 


1.74 


0.52 


Statistically 

significant 


-h20 


Pre-CTOPPP Elision subtest 
(English) 


Preschoolers 


10/63 


8.04 

(3.51) 


6.37 

(1.51) 


1.67 


0.61 


Statistically 

significant 


-h23 


Pre-CTOPPP Blending subtest 
(Spanish) 


Preschoolers 


10/63 


12.71 

(4.06) 


10.59 

(3.02) 


2.12 


0.59 


Statistically 

significant 


-h22 


Pre-CTOPPP Elision subtest 
(Spanish) 


Preschoolers 


10/63 


7.40 

(2.95) 


5.52 

(1.32) 


1.88 


0.82 


Statistically 

significant 


-h29 








Lonigan et al., 2005 (workshop group)® 










Pre-CTOPPP Blending subtest 


Preschoolers 


33/442 


14.23 

(4.34) 


13.47 

(4.47) 


0.76 


0.17 


ns 


+7 


Pre-CTOPPP Elision subtest 


Preschoolers 


33/442 


8.91 

(3.79) 


7.43 

(3.54) 


1.48 


0.40 


Statistically 

significant 


-Hi 6 



Lonigan et al., 2005 (mentoring group)^ 



Pre-CTOPPP Blending subtest 


Preschoolers 


33/516 


14.14 

(4.54) 


13.47 

(4.47) 


0.67 


0.15 


ns 


-h6 


Pre-CTOPPP Elision subtest 


Preschoolers 


33/516 


8.86 

(4.03) 


7.43 

(3.54) 


1.43 


0.37 


Statistically 

significant 


-Hi 5 



(continued) 
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Appendix A4.3 Summary of subgroup findings for the phonoiogicai processing domain (continued) 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Lonigan et al., 2005 (English speakers in the California sample)^ 



Pre-CTOPPP Blending subtest 


Preschoolers 


30/1 98^ 


14.70 

(4.49) 


14.10 

(4.12) 


0.60 


0.14 


ns 


+5 


Pre-CTOPPP Elision subtest 


Preschoolers 


30/1 98^ 


9.07 

(3,43) 


7.75 

(3.32) 


1.32 


0,39 


Statistically 

significant 


+15 






Lonigan et al., 2005 (Spanish speakers in the California sample)® 








Pre-CTOPPP Blending subtest 


Preschoolers 


30/2319 


12,94 


12.18 


0.76 


0.16 


ns 


+6 








(4.83) 


(4.38)*“ 










Pre-CTOPPP Elision subtest 


Preschoolers 


30/2319 


6.94 


6.11 


0.83 


0,27 


Statistically 


+11 








(3,40) 


(2.09) 






significant 





ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix presents subgroup findings for measures that fall in the phonological processing domain. Total group scores were used for rating purposes and are presented in Appendix A3. 3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronological age and pretest. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 

comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 

dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Farver, Lonigan, and Eppe (2009), a correction 
for clustering was not needed; however, a correction for multiple comparisons was needed, so the significance levels may differ from the original study. In the case of Lonigan et al. (2005), a 
correction for multiple comparisons was needed, so the significance levels may differ from those reported in the original study. 

9. The number of preschools in each group at assignment and the number of children in each group at posttest was provided by the study authors upon WWC request. 

10. The correct standard deviation was provided by the study authors upon WWC request. 
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Appendix A4.4 Summary of subgroup findings for the cognition domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

group group 


Mean 

difference^ 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Pre-CTOPPP Non-Word 
Repetition subtest 


Preschooiers 


33/442 


Lonigan et al., 2005 (workshop group)^ 

9.29 9.55 

(3.96) (4.36) 


-0.26 


-0.06 


ns 


-2 


Pre-CTOPPP Word Span subtest 


Preschooiers 


33/442 


8.84 

(2.42) 


8.53 

(2.54) 


0.31 


0.12 


ns 


-h5 


Pre-CTOPPP Rapid Object 
Naming subtest 


Preschooiers 


33/442 


48.51 

(20.36)“ 


49.57 

(16.93) 


1.06“ 


0.06 


ns 


+2 








Lonigan et al., 2005 (mentoring group)^ 










Pre-CTOPPP Non-Word 


Preschooiers 


33/516 


8.89 


9.55 


-0.66 


-0.15 


ns 


-6 


Repetition subtest 






(4.29) 


(4.36) 










Pre-CTOPPP Word Span subtest 


Preschooiers 


33/516 


8.88 


8.53 


0.35 


0.13 


ns 


-h5 








(2.66) 


(2.54) 










Pre-CTOPPP Rapid Object 


Preschooiers 


33/516 


47.87 


49.57 


1.70“ 


0.11 


ns 


-h4 


Naming subtest 






(14.34) 


(16.93) 














Lonigan et al., 2005 (English speakers in the California sample)^ 








Pre-CTOPPP Non-Word 


Preschooiers 


30/1981“ 


10.62 


9.55 


1.07 


0.26 


ns 


-hIO 


Repetition subtest 






(4.16) 


(3.98) 










Pre-CTOPPP Word Span subtest 


Preschoolers 


30/1981“ 


9.30 


8.57 


0.73 


0.29 


ns 


-nil 








(2.57) 


(2.41) 










Pre-CTOPPP Rapid Object 


Preschooiers 


30/1981“ 


44.89 


46.24 


1.35“ 


0.09 


ns 


-h4 


Naming subtest 






(14.90) 


(15.11) 














Lonigan et al., 2005 (Spanish speakers in the California sample)^ 








Pre-CTOPPP Non-Word Repeti- 


Preschooiers 


30/2311“ 


8.52 


8.38 


0.14 


0.04 


ns 


+2 


tion subtest 






(3.89) 


(3.59) 










Pre-CTOPPP Word Span subtest 


Preschoolers 


30/2311“ 


7.47 


7.37 


0.10 


0.04 


ns 


+2 








(2.29) 


(2.32) 










Pre-CTOPPP Rapid Object 


Preschoolers 


30/2311“ 


50.03 


47.37 


-2.66“ 


-0.16 


ns 


-6 


Naming subtest 






(18.92) 


(11.52) 











(continued) 
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Appendix A4.4 Summary of subgroup findings for the cognition domain (continued) 

ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 

1. This appendix presents subgroup findings for measures that fall in the cognition domain. Total group scores were used for rating purposes and are presented in Appendix A3. 4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no correction for 
clustering was needed and no correction for multiple comparisons was needed because no impacts were statistically significant. 

8. The correct standard deviation was provided by the study authors upon WWC request. 

9. For this outcome, the mean difference was calculated so that a positive effect was found when intervention group children took less time than comparison group children to complete the task 
(comparison group mean minus the intervention group mean). 

10. The number of preschools in each group at assignment and the number of children in each group at posttest was provided by the study authors upon WWC request. 
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Appendix A4.5 Summary of findings for comparisons between Literacy Express Engiish-oniy and Literacy Express transitionai for the orai 
ianguage domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 
English-only transitional 

groups group 


Mean 

difference'* 
{Literacy Express 
English-only - 
Literacy Express 
transitional) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Farver, Lonigan, & Eppe, 2009^ 



Pre-CTOPPP Receptive 
Vocabulary subtest (English) 


Preschoolers 


10/62 


30.62 

(5.85) 


31.79 

(3.95) 


-1.17 


-0.23 


ns 


-9 


Pre-CTOPPP Definitional 
Vocabulary subtest (English) 


Preschoolers 


10/62 


47.45 

(12,96) 


52,28 

(12.07) 


-4.83 


-0.38 


ns 


-15 


Pre-CTOPPP Receptive 
Vocabulary subtest (Spanish) 


Preschoolers 


10/62 


24.58 

(4.07) 


27.03 

(5,74) 


-2.45 


-0.49 


ns 


-19 


Pre-CTOPPP Definitional 
Vocabulary subtest (Spanish) 


Preschoolers 


10/62 


25.90 

(19,30) 


32.66 

(18,30) 


-6.76 


-0.35 


ns 


-14 



ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1 . This appendix presents subgroup findings for measures that fall in the oral language domain. Total group scores were used for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronological age and pretest. 

4. Positive differences and effect sizes favor the first group (in this case. Literacy Express English-only group); negative differences and effect sizes favor the second group (in this case. Literacy 
Express transitional group). 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Farver, Lonigan, and Eppe (2009), a correction 
for clustering was not needed; however, a correction for multiple comparisons was needed, so the significance levels may differ from the original study. 
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Appendix A4.6 Summary of findings for comparisons between Literacy Express mentoring and Literacy Express workshop for the orai 
ianguage domain^ 









Authors’ findings from the study 
















Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 

mentoring group workshop group 


Mean 

difference^ 
{Literacy Express 
mentoring - 
Literacy Express 
workshop) 


Effect 

size'* 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 



Lonigan et al., 2005^ 



PLS-IV Expressive Communica- Preschoolers 30/486 54.04 52.63 1.41 0.16 ns +6 

tion subtest (7.97) (9.68) 

ns = not statistically significant 

PLS-IV = Preschool Language Scales-IV 

1 . This appendix presents subgroup findings for measures that fall in the oral language domain. Total group scores were used for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the first group (in this case, Literacy Express mentoring group); negative differences and effect sizes favor the second group (in this case, Literacy 
Express workshop group). 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no correction for multiple 
comparisons was needed and no correction for clustering was needed because the analysis used HLM. 
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Appendix A4.7 Summary of findings for comparisons between Literacy Express Engiish-oniy and Literacy Express transitionai for the 
print knowiedge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 
English-only transitional 

groups group 


Mean 

difference'* 
{Literacy Express 
English-only - 
Literacy Express 
transitional) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Farver, Lonigan, & Eppe, 2009^ 



Pre-CTOPPP Print Knowledge 
subtest (English) 


Preschoolers 


10/62 


20.11 

(9.01) 


23.90 

(7,56) 


-3.79 


-0,45 


ns 


-17 


Pre-CTOPPP Print Knowledge 
subtest (Spanish) 


Preschoolers 


10/62 


13,14 

(6,62) 


16,54 

(8.90) 


-3,40 


-0,43 


ns 


-17 



ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix presents subgroup findings for measures that fall in the print knowledge domain. Total group scores were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronological age and pretest. 

4. Positive differences and effect sizes favor the first group (in this case. Literacy Express English-only group); negative differences and effect sizes favor the second group (in this case. Literacy 
Express transitional group). 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Farver, Lonigan, and Eppe (2009), 

no correction for clustering was needed; however, a correction for multiple comparisons was needed, so the significance levels may differ from the original study. 
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Appendix A4.8 Summary of findings for comparisons between Literacy Express mentoring and Literacy Express workshop for the print 
knowiedge domain^ 









Authors’ findings from the study 
















Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 

mentoring group workshop group 


Mean 

difference^ 
{Literacy Express 
mentoring - 
Literacy Express 
workshop) 


Effect 

size'* 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 



Lonigan et al., 2005^ 



Pre-CTOPPP Print Preschoolers 30/486 18.77 16.19 2.58 0.28 Statistically +11 

Knowledge subtest (9.16) (9.21) significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 

1. This appendix presents subgroup findings for measures that fall in the print knowledge domain. Total group scores were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the first group (in this case, Literacy Express mentoring group); negative differences and effect sizes favor the second group (in this case, Literacy 
Express workshop group). 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no correction for multiple 
comparisons was needed and no correction for clustering was needed because the analysis used HLM. 
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Appendix A4.9 Summary of findings for comparisons between Literacy Express Engiish-oniy and Literacy Express transitionai for the 

phonoiogicai processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 
English-only transitional 

groups group 


Mean 

difference'* 
{Literacy Express 
English-only - 
Literacy Express 
transitional) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



Farver, Lonigan, & Eppe, 2009^ 



Pre-CTOPPP Blending 
subtest (English) 


Preschoolers 


10/62 


14.31 

(3.33) 


14,43 

(3.04) 


-0.12 


-0,04 


ns 


-1 


Pre-CTOPPP Eiision 
subtest (English) 


Preschoolers 


10/62 


7.96 

(3.24) 


8.04 

(3.51) 


-0.08 


-0,02 


ns 


-1 


Pre-CTOPPP Blending 
subtest (Spanish) 


Preschoolers 


10/62 


11.13 

(2,49) 


12,71 

(4,06) 


-1.58 


-0,46 


ns 


-18 


Pre-CTOPPP Eiision 
subtest (Spanish) 


Preschoolers 


10/62 


5.94 

(1.75) 


7.40 

(2.95) 


-1.46 


-0,59 


ns 


-22 



ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix presents subgroup findings for measures that fall in the phonological processing domain. Total group scores were used for rating purposes and are presented in Appendix A3. 3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. In the case of Farver, Lonigan, and Eppe (2009), the means are adjusted for chronological age and pretest. 

4. Positive differences and effect sizes favor the first group (in this case. Literacy Express English-only group); negative differences and effect sizes favor the second group (in this case. Literacy 
Express transitional group). 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Farver, Lonigan, and Eppe (2009), 

no correction for clustering was needed; however, a correction for multiple comparisons was needed, so the significance levels may differ from the original study. 
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Appendix A4.10 Summary of findings for comparisons between Literacy Express mentoring and Literacy Express workshop for the 
phonoiogicai processing domain^ 









Authors’ findings from the study 
















Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 

mentoring group workshop group 


Mean 

difference^ 
{Literacy Express 
mentoring - 
Literacy Express 
workshop) 


Effect 

size'* 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 



Lonigan et al., 2005^ 



Pre-CTOPPP Blending subtest 


Preschoolers 


30/486 


14.14 

(4.54) 


14.23 

(4.34) 


-0.09 


-0.02 


ns 


-1 


Pre-CTOPPP Elision subtest 


Preschoolers 


30/486 


8.86 

(4.03) 


8.91 

(3.79) 


-0.05 


-0.01 


ns 


-1 



ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix presents subgroup findings for measures that fall in the phonological processing domain. Total group scores were used for rating purposes and are presented in Appendix A3. 3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the first group (in this case, Literacy Express mentoring group); negative differences and effect sizes favor the second group (in this case, Literacy 
Express workshop group). 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no correction for 
clustering was needed because the analysis used HLM. A correction for multiple comparisons was needed, however, so the significance levels may differ from the original study. 
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Appendix A4.11 Summary of findings for comparisons between Literacy Express mentoring and Literacy Express workshop for the 
cognition domain^ 









Authors’ findings from the study 
















Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Literacy Express 

mentoring group workshop group 


Mean 

difference^ 
{Literacy Express 
mentoring - 
Literacy Express 
workshop) 


Effect 

size'* 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 



Lonigan et al., 2005^ 



Pre-CTOPPP Non-Word 
Repetition subtest 


Preschooiers 


30/486 


8.89 

(4.29) 


9.29 

(3.96) 


-0.40 


-0.10 


ns 


-4 


Pre-CTOPPP Word Span subtest 


Preschooiers 


30/486 


8.88 

(2.66) 


8.84 

(2.42) 


0.04 


0.02 


ns 


+1 


Pre-CTOPPP Rapid Object 
Naming subtest 


Preschooiers 


30/486 


47.87 

(14.34) 


48.51 

(20.36)8 


0.649 


0.04 


ns 


+1 



ns = not statistically significant 

Pre-CTOPPP = Preschool Comprehensive Test of Phonological and Print Processing 



1. This appendix presents subgroup findings for measures that fall in the cognition domain. Total group scores were used for rating purposes and are presented in Appendix A3. 4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the first group (in this case, Literacy Express mentoring group); negative differences and effect sizes favor the second group (in this case, Literacy 
Express workshop group). 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Lonigan et al. (2005), no corrections for 
clustering or multiple comparisons were needed because the analysis corrected for clustering by using HLM and no impacts were statistically significant. 

8. The correct standard deviation was provided by the study authors upon WWC request. 

9. For this outcome, the mean difference was calculated so that a positive effect was found when intervention group children took less time than comparison group children to complete the task 
(comparison group mean minus the intervention group mean). 
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Appendix A4.12 Summary of follow-up findings for the oral language domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
(Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008^ 



PPVT-III 


Kindergarteners 


12/151 


92.09 

(15,12) 


89,23 

(12,46) 


2.86 


0.16 


ns 


-h6 


T0LD-P:3 Grammatic 
Understanding subtest 


Kindergarteners 


12/154 


8.96 

(3.42) 


8.44 

(3.25) 


0.52 


0.10 


ns 


-h4 



ns = not statistically significant 

PPVT-III = Peabody Picture Vocabulary Test-Ill 

T0LD-P:3 = Test of Language Development-Primary III 

1. This appendix presents follow-up findings for measures that fall in the oral language domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. For PCER Consortium (2008), each intervention group mean is calculated as the unadjusted control mean plus the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -r50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections for 
clustering or multiple comparisons were needed because the analysis corrected for clustering by using HLM, and no impacts were statistically significant. 
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Appendix A4.13 Summary of follow-up findings for the print knowledge domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008^ 



TERA-3 


Kindergarteners 


12/153 


81.21 

(15.03) 


82.06 

(13.38) 


-0.85 


-0.11 


ns 


-4 


WJ-III Letter-Word 
Identification subtest 


Kindergarteners 


12/151 


101.90 

(14.35) 


99.74 

(12.15) 


2.16 


0.08 


ns 


-f3 


WJ-III Spelling subtest 


Kindergarteners 


12/147 


99.38 


97.83 


1.55 


0.06 


ns 


+2 



(17.67) (13.15) 

ns = not statistically significant 

TERA-3 = Test of Early Reading Ability-Ill 

WJ-III = Woodcock-Johnson III 

1. This appendix presents follow-up findings for measures that fall in the print knowledge domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. For PCER Consortium (2008), each intervention group mean is calculated as the unadjusted control mean plus the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -r50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections for 
clustering or multiple comparisons were needed because the analysis corrected for clustering by using HLM, and no impacts were statistically significant. 
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Appendix A4.14 Summary of follow-up findings for the phonological processing domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 









PCER Consortium, 2008® 










CTOPP Elision subtest 


Kindergarteners 


12/155 


2.95 2.68 

(3.43) (2.98) 


0.27 


0.08 


ns 


-h3 



ns = not statistically significant 

CTOPP = Comprehensive Test of Phonological Processing 



1. This appendix presents follow-up findings for measures that fall in the phonological processing domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3. 3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. For PCER Consortium (2008), each intervention group mean is calculated as the unadjusted control mean plus the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -r50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Stan- 
dards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections for 
clustering or multiple comparisons were needed, because the analysis corrected for clustering by using HLM, and no impacts were statistically significant. 
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Appendix A4.15 Summary of follow-up findings for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children) 


Literacy Express Comparison 

groups group 


Mean 

difference'* 
{Literacy Express 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



PCER Consortium, 2008^ 



WJ-III Applied Problems subtest 


Kindergarteners 


12/147 


90.10 

(17.45) 


90.54 

(14.23) 


-0.44 


-0.02 


ns 


-1 


CMA-A Composite 


Kindergarteners 


12/154 


0.52 

(0.22) 


0.57 

(0.21) 


-0.05 


-0.21 


ns 


-8 


Shape Composition 


Kindergarteners 


12/153 


1.98 


2.11 


-0.13 


-0.14 


ns 


-6 



(097] (091) 

ns = not statistically significant 

WJ-III = Woodcock-Johnson III 

CMA-A = Child Math Assessment-Abbreviated 

1. This appendix presents follow-up findings for measures that fall in the math domain. End-of-preschool scores were used for rating purposes and are presented in Appendix A3. 5. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. For PCER Consortium (2008), each intervention group mean is calculated as the unadjusted control mean plus the covariate-adjusted mean difference. Standard deviations are unadjusted. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Flandbook, Appendix B. In the case of PCER Consortium (2008), the WWC used the effect sizes reported 
by the study authors. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards Flandbook, Appendix D for multiple comparisons. In the case of PCER Consortium (2008), no corrections 
clustering or multiple comparisons were needed because the analysis corrected for clustering by using HLM, and no impacts were statistically significant. 
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Appendix A5.1 Literacy Express rating for the orai ianguage domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of orai ianguage, the WWC rated Literacy Express as having positive effects for preschooi chiidren. The remaining ratings (potentially 
positive effects, mixed effects, no discernible effects, potentially negative effects, negative effects) were not considered, as Literacy Express was assigned the highest 
applicable rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Two studies that measured oral language showed a statistically significant positive effect, and one study showed no effect. All of these studies 
met WWC evidence standards for a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. None of the three studies that measured oral language showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 



Appendix A5.2 Literacy Express rating for the print knowiedge domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.'' 

For the outcome domain of print knowledge, the WWC rated Literacy Express as having positive effects for preschool children. The remaining ratings (potentially 
positive effects, mixed effects, no discernible effects, potentially negative effects, negative effects) were not considered, as Literacy Express was assigned the highest 
applicable rating. 

Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Two studies that measured print knowledge showed a statistically significant positive effect, and one study showed no effect. All of these 
studies met WWC evidence standards for a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. None of the three studies that measured print knowledge showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.3 Literacy Express rating for the phonoiogicai processing domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of phonoiogicai processing, the WWC rated Literacy Express as having positive effects for preschool children. The remaining ratings 
(potentially positive effects, mixed effects, no discernible effects, potentially negative effects, negative effects) were not considered, as Literacy Express was assigned 
the highest applicable rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Two of the studies that measured phonological processing showed a statistically significant positive effect, and one study found no effect. All 
of the studies met WWC standards for a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. None of the three studies that measured phonological processing showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 



WWC Intervention Report Literacy Express 



July 2010 



42 





Appendix A5.4 Literacy Express rating for the cognition domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of cognition, the WWC rated Literacy Express as having no discernible effects for preschool children. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: No studies showing a statistically significant or substantively important effect, either positive or negative. 

Met. The one study that measured cognition showed no statistically significant or substantively important positive or negative effects. No other 
studies measured cognition. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. The one study that measured cognition showed no statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The one study that measured cognition did not show a statistically significant or substantively important negative effect. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. The one study that measured cognition showed no statistically significant or substantively important positive effects. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. The one study that measured cognition showed no statistically significant or substantively important negative effects. No other studies 
measured cognition. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. The one study that measured cognition showed no statistically significant or substantively important positive or negative effects. No other 
studies measured cognition. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. The one study that measured cognition showed no statistically significant or substantively important positive or negative effects. No other 
studies measured cognition. 

(continued) 
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Appendix A5.4 Literacy Express rating for the cognition domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: One study showing a statistically significant or substantively important negative effect and no studies showing a statistically significant or substantively 
important positive effect. 

Not met. The one study that measured cognition showed no statistically significant or substantively important negative effects. No other studies 
measured cognition. 

OR 

• Criterion 2; Two or more studies showing statistically significant or substantively important negative effects, at least one study showing a statistically significant 
or substantively important positive effect, and more studies showing statistically significant or substantively important negative effects than showing statistically 
significant or substantively important positive effects. 

Met. The one study that measured cognition showed no statistically significant or substantively important positive effects. No other studies 
measured cognition. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. The one study that measured cognition showed no statistically significant negative effects. No other studies measured cognition. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. The one study that measured cognition showed no statistically significant or substantively important positive effects. No other studies 
measured cognition. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.5 Literacy Express rating for the math domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math, the WWC rated Literacy Express as having no discernibie effects for preschooi chiidren. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: No studies showing a statisticaily significant or substantively important effect, either positive or negative. 

Met. The one study that measured math showed no statisticaily significant or substantiveiy important positive or negative effects. No other studies 
measured math. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. The one study that measured math showed no statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. The one study that measured math did not show a statistically significant or substantiveiy important negative effect. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect. 

Not met. The one study that measured math showed no statistically significant or substantively important positive effects. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. The one study that measured math showed no statistically significant or substantively important negative effects. No other studies 
measured math. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. The one study that measured math showed no statistically significant or substantively important positive or negative effects. No other 
studies measured math. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. The one study that measured math showed no statistically significant or substantively important positive or negative effects. No other 
studies measured math. 

(continued) 
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Appendix A5.5 Literacy Express rating for the math domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: One study showing a statistically significant or substantively important negative effect and no studies showing a statistically significant or substantively 
important positive effect. 

Not met. The one study that measured math showed no statistically significant or substantively important negative effects. No other studies 

measured math. 

OR 

• Criterion 2; Two or more studies showing statistically significant or substantively important negative effects, at least one study showing a statistically significant 
or substantively important positive effect, and more studies showing statistically significant or substantively important negative effects than showing statistically 
significant or substantively important positive effects. 

Met. The one study that measured math showed no statistically significant or substantively important positive effects. No other studies measured math. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. The one study that measured math showed no statistically significant negative effects. No other studies measured math. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. The one study that measured math showed no statistically significant or substantively important positive effects. No other studies measured math. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Sample size 
Preschool classrooms 


Students 


Extent of evidence^ 


Oral language 


3 


70 




1,002 


Medium to large 


Print knowledge 


3 


70 




999 


Medium to large 


Phonological processing 


3 


70 




1,004 


Medium to large 


Early reading and writing 


0 


na 




na 


na 


Cognition 


1 


48 




722 


Small 


Math 


1 


12 




185 


Small 



na = not applicable/not studied 

1. A rating of “medium to iarge” requires at ieast two studies and two schoois across studies in one domain and a totai sampie size across studies of at ieast 350 students or 14 ciassrooms. 
Otherwise, the rating is “smaii.” For more detaiis on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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